Video image de-interlacing method and video image de-interlacing device

ABSTRACT

A video image de-interlacing method is provided. The method-includes: acquiring a single frame of original video image; extracting odd field data and even field data in an original video image; performing N-1 times of down-sampling on the odd field data to obtain N-1 odd field data with different resolutions and performing N-1 times of down-sampling on the even field data to obtain N-1 even field data with different resolutions; combining odd field data and even field data with the same resolution to obtain a down-sampled image; and inputting the original video image and the down-sampled image to the de-interlacing network for de-interlacing.

TECHNICAL FIELD

The embodiments of the present disclosure relate to the technical fieldof image processing, and more particularly relate to a video imagede-interlacing method and a video image de-interlacing device.

BACKGROUND

In the related art, for a progressive scanning display device, if areceived video image is an interlaced video image, the interlaced videoimage needs to be de-interlaced to obtain a progressive scanning videoimage. When de-interlacing interlaced video images, how to improve ade-interlacing effect is an urgent problem to be solved.

SUMMARY

In a first aspect, embodiments of the present disclosure provide a videoimage de-interlacing method which includes:

-   acquiring a single frame original video image including parity field    information;-   extracting odd field data and even field data in the original video    image;-   performing N-1 times of down-sampling on the odd field data to    obtain N-l odd field data with different resolutions, and performing    N-l times of down-sampling on the even field data to obtain N-l even    field data with different resolutions; combining odd field data and    even field data with the same resolution in N-l odd field data with    different resolutions and N-l even field data with different    resolutions to obtain N-l down-sampled images with different    resolutions;-   inputting an image with N resolutions which includes the original    video image and the down-sampled image with N-l resolutions into a    de-interlacing network to perform de-interlacing processing so as to    obtain a de-interlaced image in the image with N resolutions, the    resolutions from the image with the N^(th) resolution to the image    with the first resolution increase gradually, the de-interlacing    network includes N series-connected de-interlacing sub-networks, and    the images processed by the N series-connected de-interlacing    sub-networks are respectively generated based on the image with N    resolutions, wherein-   N is a positive integer greater than or equal to 2.

Optionally, inputting the image with the N resolutions including theoriginal video image and the down-sampled image with N-l resolutionsinto the de-interlacing network to perform the de-interlacing processingso as to obtain the de-interlaced image includes:

-   splicing two down-sampled images with the N^(th) resolution to    obtain a spliced image with the N^(th) resolution; inputting the    N^(th) resolution spliced image into an N^(th) serial de-interleaved    sub-network to obtain an N^(th) resolution de-interleaved image;    up-sampling the N^(th) resolution de-interlaced image to obtain an    N-1^(th) resolution up-sampled image;-   splicing an up-sampled image with an i^(th) resolution and a    down-sampled image with the i^(th) resolution to obtain a spliced    image with the i^(th) resolution; inputting the spliced image with    the i^(th) resolution into an i^(th) deinterleaving sub-network to    obtain a de-interleaved image with the i^(th) resolution; performing    up-sampling processing on the de-interlaced image with the i^(th)    resolution to obtain an up-sampled image with an i-1^(th)    resolution; wherein i is an integer greater than or equal to 2 and    less than N; and-   splicing an up-sampled image with a first resolution and an original    video image with the first resolution to obtain a spliced image with    the first resolution; inputting the first resolution spliced image    into a first de-interlacing network to obtain a first resolution    de-interlaced image as an output image of the de-interlacing    network.

Optionally, combining the odd field data and the even field data withthe same resolution among the odd field data with N-l differentresolutions and the even field data with N-l different resolutionsincludes:

arranging the odd field data and the even field data having the sameresolution in line intervals.

Optionally, the N is 3 or 4.

Optionally, the resolutions of adjacent images with the N resolutionshave a relationship of 2 times.

Optionally, a bi-cubic interpolation method is used for up-samplingand/or down-sampling.

Optionally, N series-connected de-interlacing sub-networks have the samestructure and different parameters.

Optionally, each of the de-interlacing sub-networks includes a pluralityof series-connected filters, each filter includes a plurality ofseries-connected convolution kernels, each two of the plurality ofseries-connected filters have the same resolution, and the output ofeach filter except for the last filter serves as the input of the nextfilter and the filter with the same resolution.

Optionally, the method further includes:

-   training the de-interleaving network to be trained to obtain the    de-interleaving network,-   wherein the training of the de-interleaving network to be trained    includes:-   acquiring a single frame training video image containing parity    field information;-   extracting odd field data and even field data for training in the    video image for training;-   performing N-l times of down-sampling on the training odd field data    to obtain N-l odd field data with different resolutions, and    performing N-l times of down-sampling on the training even field    data to obtain N-l even field data with different resolutions;    combining the odd field data and the even field data with the same    resolution in N-l odd field data with different resolutions and N-l    even field data with different resolutions to obtain N-l    down-sampled images for training with different resolutions;-   inputting a training image with N resolutions containing the video    image for training and the down-sampled image for training with N-l    resolutions into the de-interlacing network to be trained to perform    de-interlacing processing so as to obtain output images with N    resolutions, wherein in the images with N resolutions, the    resolutions from the image with the N^(th) resolution to the image    with the first resolution increase gradually, the de-interlacing    network to be trained includes N series-connected de-interlacing    sub-networks, and the images processed by the N series-connected    de-interlacing sub-networks are respectively generated based on the    images with the N resolutions; and-   calculating the loss of the N resolution output images, calculating    the total loss of the de-interlacing network to be trained according    to the loss of the N resolution output images, and optimizing the    parameters of the de-interlacing network to be trained according to    the total loss to obtain the trained de-interlacing network.

Optionally, the loss is an L2 loss.

Optionally, the total loss is equal to the sum of the losses of theoutput images with the N resolutions or a weighted sum of the losses ofthe output images with the N resolutions.

In a second aspect, embodiments of the present disclosure provide avideo image de-interlacing device which includes:

-   a first acquisition module, which is adapted for acquiring a single    frame original video image including parity field information;-   a first extraction module, which is adapted for extracting odd field    data and even field data in the original video image;-   a first down-sampling module, which is adapted for performing N-l    times of down-sampling on the odd field data to obtain N-l odd field    data with different resolutions, and performing N-l times of    down-sampling on the even field data to obtain N-l even field data    with different resolutions; combining odd field data and even field    data with the same resolution in N-l odd field data with different    resolutions and N-l even field data with different resolutions to    obtain N-l down-sampled images with different resolutions; and-   a de-interlacing module, which is adapted for inputting an image    with N resolutions comprising the original video image and the    down-sampled image with N-l resolutions into a de-interlacing    network to perform de-interlacing processing so as to obtain a    de-interlaced image in the image with N resolutions, the resolutions    from the image with the N^(th) resolution to the image with the    first resolution increase gradually, the de-interlacing network    comprises N series-connected de-interlacing sub-networks, and the    images processed by the N series-connected de-interlacing    sub-networks are respectively generated based on the image with N    resolutions, wherein-   N is a positive integer greater than or equal to 2.

In a third aspect, embodiments of the present disclosure provide anelectronic device including a processor, a memory, and a program orinstructions stored on the memory and executable on the processor, whichwhen executed by the processor performs the steps of the video imagede-interlacing method of the above first aspect.

In a fourth aspect, embodiments of the present disclosure provide anon-transitory computer-readable storage medium having stored thereon aprogram or instructions which, when executed by a processor, performsthe steps of the video image de-interlacing method in the first aspect.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a schematic flow diagram of a video image de-interlacingmethod according to an embodiment of the present disclosure;

FIG. 2 is a schematic diagram of extracting odd field data and evenfield data from an original video image according to an embodiment ofthe present disclosure;

FIGS. 3 and 4 are schematic diagrams of a method for down-sampling animage according to an embodiment of the present disclosure;

FIG. 5 is a schematic diagram of a de-interlacing network according toan embodiment of the present disclosure;

FIG. 6 is a schematic structural diagram of a de-interlacing sub-networkaccording to an embodiment of the present disclosure;

FIG. 7 is a flow diagram of a de-interlacing network training methodaccording to an embodiment of the present disclosure;

FIG. 8 is a schematic diagram of a method for calculating the total lossof a de-interlacing network according to an embodiment of the presentdisclosure;

FIG. 9 is a schematic structural diagram of a video image de-interlacingdevice according to an embodiment of the present disclosure; and

FIG. 10 is a schematic structural diagram of an electronic deviceaccording to an embodiment of the present disclosure.

DETAILED DESCRIPTION

The technical solution of embodiments of the present disclosure will nowbe described more fully hereinafter with reference to the accompanyingdrawings, in which some, but not all embodiments of the disclosure areshown. Based on the embodiments of the present disclosure, all otherembodiments obtained by a person of ordinary skill in the art withoutinventive effort fall within the scope of protection of this disclosure.

With reference to FIG. 1 , the embodiments of the present disclosureprovide a video image de-interlacing method, which includes:

Step 11: acquiring a single frame original video image including parityfield information.

Here, the original video image is a video image obtained by aninterlacing scanning means, wherein the electron beam first scans allthe odd lines from left to right and from top to bottom to form onefield of image data, and then the electron beam returns to the top andscans all the even lines from left to right and from top to bottom toform another field of image data. The scan fields displayed by the twoorthogonal directions are exchanged to form each complete video imagefor each frame.

Step 12: extracting the odd field data and the even field data in theoriginal video imag.

Referring to FIG. 2 , wherein FIG. 2 is a schematic diagram ofextracting the odd field data and the even field data from the originalvideo image according to an embodiment of the present disclosure.

Step 13: performing N-l times of down-sampling on the odd field data toobtain N-l odd field data with different resolutions, and performing N-ltimes of down-sampling on the even field data to obtain N-l even fielddata with different resolutions; combining odd field data and even fielddata with the same resolution in N-l odd field data with differentresolutions and N-l even field data with different resolutions to obtainN-l down-sampled images with different resolutions.

For example, the odd field data and the even field data of the originalvideo image are respectively down-sampled twice. Please refer to FIGS. 3and 4 , and the down-sampling times are respectively 2 times and 4times. Assuming that the resolutions of the odd field data and the evenfield data are both 256 × 256, down-sampling twice results in the oddfield data and the even field data with a resolution of 128×128, anddown-sampling 4 times results in the odd field data and the even fielddata with a resolution of 64×64.

In the disclosed embodiment, the odd field data and the even field dataafter down-sampling can be combined in a concatenation manner of the oddfield data and the even field data in the original video image.

Optionally, referring to FIGS. 3 and 4 , arranging the odd field dataand the even field data having the same resolution in line intervals,i.e., interlacing and combining.

Step 14: inputting an image with the N resolutions including theoriginal video image and the down-sampled image with N-l resolutionsinto the de-interlacing network for de-interlacing processing to obtainthe de-interlaced image, wherein in the image with N resolutions, theresolutions from the image with the N^(th) resolution to the image withthe first resolution increase gradually, the de-interlacing networkincludes N series de-interlacing sub-networks, and the images processedby the N series de-interlacing sub-networks are respectively generatedbased on the image with N resolutions.

Specifically, N is a positive integer greater than or equal to 2.

In the embodiment of the present disclosure, the N de-interlacingnetworks are used for processing images with one of the N resolutions.

In the embodiment of the present disclosure, a final outputde-interlaced image from the de-interlacing network is a progressivescanning video image.

In the embodiment of the present disclosure, down-sampling the extractedodd field data and the even field data of the original video image tothe plurality of resolutions and performing de-interlacing at theplurality of resolutions can ensure that the information about theoriginal video image is not destroyed when down-sampling, and performingprogressive de-interlacing can achieve a better de-interlacing effect.

The operation of the de-interlacing network of embodiments of thepresent disclosure is described below.

In an embodiment of the present disclosure, optionally, inputting theimage with the N resolutions including the original video image and thedown-sampled image with N-l resolutions into the de-interlacing networkto perform the de-interlacing processing so as to obtain thede-interlaced image includes:

-   performing the following steps on the minimum resolution image:    splicing two down-sampled images with the N^(th) resolution to    obtain a spliced image with the N^(th) resolution; inputting the    N^(th) resolution spliced image into an N^(th) serial de-interleaved    sub-network to obtain an N^(th) resolution de-interleaved image;    up-sampling the N^(th) resolution de-interlaced image to obtain an    N-1^(th) resolution up-sampled image;-   performing the following steps on images with intermediate    resolution: splicing an up-sampled image with an i^(th) resolution    and a down-sampled image with the i^(th) resolution to obtain a    spliced image with the i^(th) resolution; inputting the spliced    image with the i^(th) resolution into an i^(th) de-interleaving    sub-network to obtain a de-interleaved image with the i^(th)    resolution; performing up-sampling processing on the de-interlaced    image with the i^(th) resolution to obtain an up-sampled image with    an i-1^(th) resolution; wherein i is an integer greater than or    equal to 2 and less than N; and-   performing the following steps on the maximum resolution image:    splicing an up-sampled image with a first resolution and an original    video image with the first resolution to obtain a spliced image with    the first resolution; inputting the first resolution spliced image    into a first de-interlacing network to obtain a first resolution    de-interlaced image as an output image of the de-interlacing    network.

In the embodiments of the present disclosure, when de-interleaving isperformed on the video image, starting from a low resolution,continuously increasing the resolution and performing progressivede-interleaving so as to achieve a better de-interleaving effect, andsince odd field data and even field data of an extracted original videoimage are respectively down-sampled during down-sampling, it can beensured that information about the original video image is not destroyedduring down-sampling.

In the embodiment of the present disclosure, optionally, the N is 3 or4, so as to achieve the better de-interlacing effect and effectivelyreduce the implementation cost. Of course, other values of N greaterthan or equal to 2 are not excluded.

In the embodiment of the present disclosure, optionally, the resolutionsof adjacent resolution images in the N resolution images are in a 2times relationship, for example, the resolutions of the N resolutionimages are 256×256, 128×128 and 64×64 respectively.

In the embodiment of the present disclosure, optionally, the bi-cubicinterpolation method is used for up-sampling and/or down-sampling topreserve better image details. Of course, other interpolation methodsmay be used for up-sampling and/or down-sampling, such as bilinearinterpolation, etc.

With reference to FIG. 5 , FIG. 5 is a schematic diagram of thede-interlacing network according to the embodiment of the presentdisclosure, wherein the de-interlacing network includes threede-interlacing sub-networks: a de-interlacing sub-network 1, ade-interlacing sub-network 2, and a de-interlacing sub-network 3.

The working process of a de-interlacing network is as follows:

-   1) extracting the odd field data and the even field data in the    original video image;-   2) down-sampling the odd field data and the even field data of the    original video image twice, respectively 2 times and 4 times, so as    to obtain down-sampled images, Down_x2_(odd) , Down_x2_(even) ,    Down_x4_(odd) , and Down_x4_(even) of the odd field data and the    even field data, and combining the Down_x2_(odd) and Down_x2_(even)    to obtain the Down_x2 , and combining the Down_x4_(odd)and    Down_x4_(even) to obtain the Down_x4;-   3) splicing the two Down_x4 to obtain a spliced image, and inputting    the spliced image into a de-interlacing sub-network 3 to perform    de-interlacing processing to obtain a de-interlaced image;-   4) up-sampling the de-interlaced image out_x4 by 2 times to obtain    Up_x2, splicing theUp_x2andDown_x2to obtain a spliced image, and    inputting the spliced image into a de-interlacing sub-network 2 to    perform de-interlacing processing to obtain a de-interlaced    imageout_x2; and-   5) up-sampling the de-interlaced image out_x2 by 2 times to obtain    Up_x1, and splicing the Up_x1and the original vedio image to obtain    a splicing image, and inputting the splicing image into the    de-interlacing sub-network 1 to perform de-interlacing processing to    obtain a final output.

In the embodiment of the present disclosure, optionally, the Nconcatenated de-interlacing sub-networks have the same structure anddifferent parameters.

With reference to FIG. 6 , FIG. 6 is a schematic structural diagram ofthe de-interlacing sub-network according to the embodiment of thepresent disclosure, and the de-interlacing sub-network includes: aplurality of filters connected in series and each filter includes aplurality of convolution kernels connected in series (vertical bars inFIG. 6 ). In the embodiment shown in FIG. 6 , each filter includes fourconvolution kernels in series, although in other embodiments of thepresent disclosure, the number of convolution kernels in the filter isnot limited to four. In the embodiment shown in FIG. 6 , the verticalbars filled with twill indicate down-sampling and the vertical barsfilled with dots indicate up-sampling. In the embodiment of the presentdisclosure, each of the de-interlacing sub-networks includes a pluralityof series-connected filters, each filter includes a plurality ofseries-connected convolution kernels, each two of the plurality ofseries-connected filters have the same resolution, and the output ofeach filter except for the last filter serves as the input of the nextfilter and the filter with the same resolution. In the embodiment shownin FIG. 6 , the de-interlacing sub-network includes six filtersconnected in series, wherein the resolution of the first filter is thesame as that of the sixth filter, the resolution of the second filter isthe same as that of the fifth filter, the resolution of the third filteris the same as that of the fourth filter, the output of the first filterserves as the input of the second filter and the sixth filter (with thesame resolution as that of the first filter), and the output of thesecond filter serves as the input of the third filter and the fifthfilter (with the same resolution as that of the second filter), theoutput of the third filter serves as the input to the fourth filter(with the same resolution as that of the third filter).

In the embodiment of the present disclosure, optionally, the video imagede-interlacing method further includes: training the de-interleavingnetwork to be trained to obtain the de-interleaving network. Withreference to FIG. 7 , training the de-interleaved network to be trainedincludes:

-   Step 71: acquiring a single frame training video image containing    parity field information;-   Step 72: extracting odd field data and even field data for training    in the video image for training;-   Step 73: performing N-l times of down-sampling on the training odd    field data to obtain N-l odd field data with different resolutions,    and performing N-l times of down-sampling on the training even field    data to obtain N-l even field data with different resolutions;    combining the odd field data and the even field data with the same    resolution in N-l odd field data with different resolutions and N-l    even field data with different resolutions to obtain N-l    down-sampled images for training with different resolutions;-   Step 74: inputting a training image with N resolutions containing    the video image for training and the down-sampled image for training    with N-l resolutions into the de-interlacing network to be trained    to perform de-interlacing processing so as to obtain output images    with N resolutions, wherein in the images with N resolutions, the    resolutions from the image with the N^(th) resolution to the image    with the first resolution increase gradually, the de-interlacing    network to be trained includes N series-connected de-interlacing    sub-networks, and the images processed by the N series-connected    de-interlacing sub-networks are respectively generated based on the    images with the N resolutions; and-   Step 75: calculating the loss of the N resolution output images,    calculating the total loss of the de-interlacing network to be    trained according to the loss of the N resolution output images, and    optimizing the parameters of the de-interlacing network to be    trained according to the total loss to obtain the trained    de-interlacing network.

In the embodiment of the present disclosure, optionally, the loss is anL2 loss. Of course, other types of losses are possible.

In the embodiment of the present disclosure, optionally, the total lossis equal to the sum of the losses of the output images with the Nresolutions to a weighted sum of the losses of the N resolution outputimages. The loss of each output image is obtained based on the outputimage and the corresponding true value image of the output image.

Taking the training of the de-interlacing network in FIG. 5 as anexample, referring to FIG. 8 , the total loss of the de-interlacingnetwork is equal to the sum of the losses of the output images Output,out_x2, and Out_x4 of the three de-interlacing sub-networks, or is equalto the weighted sum of the losses of the output images Output, out_x2,and Out_x4 of the three de-interlacing sub-networks, the sum of thetotal losses is calculated, and the parameters of the de-interlacingnetwork are updated according to the calculated total losses.

With reference to FIG. 9 , the embodiment of the present disclosure alsoprovides the video image de-interlacing device 90, which includes:

-   a first acquisition module 91, which is adapted for acquiring a    single frame of original video image containing parity field    information;-   a first extraction module 92, which is adapted for extracting the    odd field data and the even field data in the original video image;-   a first down-sampling module 93, which is adapted for performing N-l    times of down-sampling on the odd field data to obtain N-l odd field    data with different resolutions, and performing N-l times of    down-sampling on the even field data to obtain N-l even field data    with different resolutions; combining odd field data and even field    data with the same resolution in N-l odd field data with different    resolutions and N-l even field data with different resolutions to    obtain N-l down-sampled images with different resolutions; and-   a de-interlacing module 94, which is adapted for inputting an image    with N resolutions including the original video image and the    down-sampled image with N-l resolutions into the de-interlacing    network for de-interlacing processing to obtain a de-interlaced    image, wherein in the image with N resolutions, the resolutions from    the image with the N^(th) resolution to the image with the first    resolution increase gradually, the de-interlacing network includes N    series-connected de-interlacing sub-networks, and the images    processed by the N series-connected de-interlacing sub-networks are    respectively generated based on the image with N resolutions.

Specifically, N is a positive integer greater than or equal to 2.

In the embodiment of the present disclosure, down-sampling the extractedodd field data and the even field data of the original video image tothe plurality of resolutions and performing de-interlacing at theplurality of resolutions can ensure that the information about theoriginal video image is not destroyed when down-sampling, and performingprogressive de-interlacing can achieve the better de-interlacing effect.

In the embodiment of the present disclosure, optionally, thede-interlacing module 94 includes:

-   a first de-interlacing sub-module, which is adapted for splicing two    down-sampled images with the N^(th) resolution; inputting the N^(th)    resolution spliced image into an N^(th) serial de-interleaved    sub-network to obtain an N^(th) resolution de-interleaved image;    up-sampling the N^(th) resolution de-interlaced image to obtain an    N-1^(th) resolution up-sampled image;-   a second de-interlacing sub-module, which is adapted for splicing    the up-sampled image with the i^(th) resolution and a down-sampled    image of the i^(th) resolution to obtain a spliced image of the    i^(th) resolution; inputting the spliced image of the i^(th)    resolution into an i^(th) deinterleaving sub-network to obtain a    de-interleaved image of the i^(th) resolution; performing    up-sampling processing on the de-interlaced image of the i^(th)    resolution to obtain an up-sampled image with an i-1^(th)    resolution, wherein i is an integer greater than or equal to 2 and    less than N; and-   a third de-interlacing sub-module, which is adapted for splicing an    up-sampled image with a first resolution and an original video image    with the first resolution to obtain a spliced image with the first    resolution; inputting the first resolution stitched image into a    first de-interlacing network to obtain a first resolution    de-interlaced image as an output image of the de-interlacing    network.

In the embodiments of the present disclosure, optionally, the N is 3 or4.

In the embodiments of the present disclosure, optionally, theresolutions of images of adjacent resolutions in the N resolution imagesare in a 2 times relationship.

In the embodiments of the present disclosure, up-sampling and/ordown-sampling is optionally performed by using a bicubic interpolationmethod.

In the embodiment of the present disclosure, optionally, the Nconcatenated de-interlacing sub-networks have the same structure anddifferent parameters.

In the embodiments of the present disclosure, optionally, each of thede-interlacing sub-networks includes a plurality of series-connectedfilters, each filter includes a plurality of series-connectedconvolution kernels, each two of the plurality of series-connectedfilters have the same resolution, and the output of each filter exceptfor the last filter serves as the input of the next filter and thefilter with the same resolution.

In an embodiment of the present disclosure, optionally, the video imagede-interleaving device further includes:

-   a training module, which is adapted for training the de-interleaving    network to be trained to obtain the de-interleaving network,-   wherein the training of the de-interleaving network to be trained    includes:-   acquiring a single frame training video image containing parity    field information;-   extracting odd field data and even field data for training in the    video image for training;-   performing N-l times of down-sampling on the training odd field data    to obtain N-l odd field data with different resolutions, and    performing N-l times of down-sampling on the training even field    data to obtain N-l even field data with different resolutions;    combining the odd field data and the even field data with the same    resolution in N-l odd field data with different resolutions and N-l    even field data with different resolutions to obtain N-l    down-sampled images for training with different resolutions;-   inputting a training image with N resolutions containing the video    image for training and the down-sampled image for training with N-l    resolutions into the de-interlacing network to be trained to perform    de-interlacing processing so as to obtain output images with N    resolutions, wherein in the images with N resolutions, the    resolutions from the image with the N^(th) resolution to the image    with the first resolution increase gradually, the de-interlacing    network to be trained includes N series-connected de-interlacing    sub-networks, and the images processed by the N series-connected    de-interlacing sub-networks are respectively generated based on the    images with the N resolutions; and-   calculating the loss of the N resolution output images, calculating    the total loss of the de-interlacing network to be trained according    to the loss of the N resolution output images, and optimizing the    parameters of the de-interlacing network to be trained according to    the total loss to obtain the trained de-interlacing network.

In the embodiment of the present disclosure, optionally, the loss is anL2 loss.

In the embodiment of the present disclosure, optionally, the total lossis equal to the sum of the losses of the output images with the Nresolutions to a weighted sum of the losses of the N resolution outputimages.

As shown in FIG. 10 , the embodiment of the present application alsoprovides an electronic device 100 which includes a processor 101, amemory 102, and a program or instruction stored in the memory 102 andoperable on the processor 101, which program or instruction, whenexecuted by the processor 101, implements the various processes of theabove-mentioned embodiment of the video image de-interlacing method, andcan achieve the same technical effect, and in order to avoid repetition,the description thereof will not be repeated.

The embodiments of the present disclosure provide a non-volatilecomputer-readable storage medium having stored thereon a program orinstructions which, when executed by a processor, performs the steps ofthe video image de-interlacing method. Each step of the video imagede-interlacing method in above-mentioned embodiments is completed, whenthe program or instructions are executed by the processor, the sametechnical effects can be achieved, and in order to avoid repetition, thedescription thereof will not be repeated.

Specifically the processor is a processor in the terminal described inthe above-mentioned embodiment. The non-volatile computer-readablestorage medium is a computer read-only Memory (ROM), a Random AccessMemory (RAM), a magnetic or optical disk, etc.

The embodiments of the present disclosure have been described above withreference to the accompanying drawings, but the present disclosure isnot limited to the above-described embodiments, which are merelyillustrative and not restrictive, and those skilled in the art, in lightof the present disclosure, can make various changes in form and detailwithout departing from the spirit and scope of the present disclosure.

1. A video image de-interlacing method, comprising: acquiring a singleframe original video image including parity field information;extracting odd field data and even field data in the original videoimage; performing N-1 times of down-sampling on the odd field data toobtain N-1 odd field data with different resolutions, and performing N-1times of down-sampling on the even field data to obtain N-1 even fielddata with different resolutions; combining odd field data and even fielddata with the same resolution in N-1 odd field data with differentresolutions and N-1 even field data with different resolutions to obtainN-1 down-sampled images with different resolutions; and inputting animage with N resolutions which comprises the original video image andthe down-sampled image with N-1 resolutions into a de-interlacingnetwork to perform de-interlacing processing so as to obtain ade-interlaced image in the image with N resolutions, the resolutionsfrom the image with an N^(th) resolution to the image with a firstresolution increase gradually, the de-interlacing network comprises Nseries-connected de-interlacing sub-networks, and the images processedby the N series-connected de-interlacing sub-networks are respectivelygenerated based on the image with N resolutions, wherein N is a positiveinteger greater than or equal to
 2. 2. The method according to claim 1,wherein inputting the image with the N resolutions comprising theoriginal video image and the down-sampled image with N-1 resolutionsinto the de-interlacing network to perform the de-interlacing processingso as to obtain the de-interlaced image comprises: splicing twodown-sampled images with the N^(th) resolution to obtain a spliced imagewith the N^(th) resolution; inputting the N^(th) resolution splicedimage into an N^(th) serial de-interleaved sub-network to obtain anN^(th) resolution de-interleaved image; up-sampling an N^(th) resolutionde-interlaced image to obtain an N-1^(th) resolution up-sampled image;splicing an up-sampled image with an i^(th) resolution and adown-sampled image with the i^(th) resolution to obtain a spliced imagewith the i^(th) resolution; inputting the spliced image with the i^(th)resolution into an i^(th) deinterleaving sub-network to obtain ade-interleaved image with the i^(th) resolution; performing up-samplingprocessing on the de-interlaced image with the i^(th) resolution toobtain an up-sampled image with an i-1^(th) resolution; wherein i is aninteger greater than or equal to 2 and less than N; and splicing anup-sampled image with a first resolution and an original video imagewith the first resolution to obtain a spliced image with the firstresolution; inputting the first resolution spliced image into a firstde-interlacing network to obtain a first resolution de-interlaced imageas an output image of the de-interlacing network.
 3. The methodaccording to claim 1, wherein combining the odd field data and the evenfield data with the same resolution among the odd field data with N-1different resolutions and the even field data with N-1 differentresolutions comprises: arranging the odd field data and the even fielddata having the same resolution in line intervals.
 4. The methodaccording to claim 1, wherein N is 3 or
 4. 5. The method according toclaim 1, wherein the resolutions of adjacent images with the Nresolutions have a relationship of 2 times.
 6. The method according toclaim 2, wherein a bi-cubic interpolation method is adopted forup-sampling and/or down-sampling.
 7. The method according to claim 1,wherein the N series-connected de-interlacing sub-networks have the samestructure and different parameters.
 8. The method according to claim 1,wherein a respective one of the de-interlacing sub-networks comprises aplurality of series-connected filters, a respective filter comprises aplurality of series-connected convolution kernels, respective twoseries-connected filters of the plurality of series-connected filtershave the same resolution, and an output of the respective filter exceptfor the last filter serves as the input of the next filter and thefilter with the same resolution.
 9. The method according to claim 1,further comprising: training a de-interleaving network to be trained toobtain the de-interleaving network, wherein the training of thede-interleaving network to be trained comprises: acquiring a singleframe training video image containing parity field information;extracting odd field data and even field data for training in the videoimage for training; performing N-1 times of down-sampling on thetraining odd field data to obtain N-1 odd field data with differentresolutions, and performing N-1 times of down-sampling on the trainingeven field data to obtain N-1 even field data with differentresolutions; combining the odd field data and the even field data withthe same resolution in N-1 odd field data with different resolutions andN-1 even field data with different resolutions to obtain N-1down-sampled images for training with different resolutions; inputting atraining image with N resolutions containing the video image fortraining and the down-sampled image for training with N-1 resolutionsinto the de-interlacing network to be trained to perform de-interlacingprocessing so as to obtain output images with N resolutions, wherein inthe images with N resolutions, the resolutions from the image with theN^(th) resolution to the image with the first resolution increasegradually, the de-interlacing network to be trained includes Nseries-connected de-interlacing sub-networks, and the images processedby the N series-connected de-interlacing sub-networks are respectivelygenerated based on the images with the N resolutions; and calculating aloss of the N resolution output images, calculating a total loss of thede-interlacing network to be trained according to the loss of the Nresolution output images, and optimizing parameters of thede-interlacing network to be trained according to the total loss toobtain the trained de-interlacing network.
 10. The method according toclaim 9, wherein the loss is an L2 loss.
 11. The method according toclaim 9, wherein the total loss is equal to a sum of the losses of theoutput images with the N resolutions or a weighted sum of the losses ofthe output images with the N resolutions.
 12. (canceled)
 13. Anelectronic device, comprising a processor, a memory, and a program orinstructions stored on the memory and executable on the processor, whenthe program or instructions are executed by the processor, a video imagede-interlacing method is realized, the method comprising: acquiring asingle frame original video image including parity field information;extracting odd field data and even field data in the original videoimage; performing N-1 times of down-sampling on the odd field data toobtain N-1 odd field data with different resolutions, and performing N-1times of down-sampling on the even field data to obtain N-1 even fielddata with different resolutions; combining odd field data and even fielddata with the same resolution in N-1 odd field data with differentresolutions and N-1 even field data with different resolutions to obtainN-1 down-sampled images with different resolutions; and inputting animage with N resolutions which comprises the original video image andthe down-sampled image with N-1 resolutions into a de-interlacingnetwork to perform de-interlacing processing so as to obtain ade-interlaced image in the image with N resolutions, the resolutionsfrom the image with an N^(th) resolution to the image with a firstresolution increase gradually, the de-interlacing network comprises Nseries-connected de-interlacing sub-networks, and the images processedby the N series-connected de-interlacing sub-networks are respectivelygenerated based on the image with N resolutions, wherein N is a positiveinteger greater than or equal to
 2. 14. A non-transitorycomputer-readable storage medium, wherein the non-transitorycomputer-readable storage medium storing program or instructions, whenthe program or the instructions are executed by one or more processors avideo image de-interlacing method is realized, the method comprising:acquiring a single frame original video image including parity fieldinformation; extracting odd field data and even field data in theoriginal video image; performing N-1 times of down-sampling on the oddfield data to obtain N-1 odd field data with different resolutions, andperforming N-1 times of down-sampling on the even field data to obtainN-1 even field data with different resolutions; combining odd field dataand even field data with the same resolution in N-1 odd field data withdifferent resolutions and N-1 even field data with different resolutionsto obtain N-1 down-sampled images with different resolutions; andinputting an image with N resolutions which comprises the original videoimage and the down-sampled image with N-1 resolutions into ade-interlacing network to perform de-interlacing processing so as toobtain a de-interlaced image in the image with N resolutions, theresolutions from the image with an N^(th) resolution to the image with afirst resolution increase gradually, the de-interlacing networkcomprises N series-connected de-interlacing sub-networks, and the imagesprocessed by the N series-connected de-interlacing sub-networks arerespectively generated based on the image with N resolutions, wherein Nis a positive integer greater than or equal to
 2. 15. The methodaccording to claim 2, wherein combining the odd field data and the evenfield data with the same resolution among the odd field data with N-1different resolutions and the even field data with N-1 differentresolutions comprises: arranging the odd field data and the even fielddata having the same resolution in line intervals.
 16. The methodaccording to claim 2, wherein N is 3 or
 4. 17. The method according toclaim 2, wherein the resolutions of adjacent images with the Nresolutions have a relationship of 2 times.
 18. The method according toclaim 2, wherein the N series-connected de-interlacing sub-networks havethe same structure and different parameters.
 19. The method according toclaim 2, wherein a respective one of the de-interlacing sub-networkscomprises a plurality of series-connected filters, a respective filtercomprises a plurality of series-connected convolution kernels,respective two series-connected filters of the plurality ofseries-connected filters have the same resolution, and the output of therespective filter except for the last filter serves as the input of thenext filter and the filter with the same resolution.
 20. The electronicdevice according to claim 13, wherein inputting the image with the Nresolutions comprising the original video image and the down-sampledimage with N-1 resolutions into the de-interlacing network to performthe de-interlacing processing so as to obtain the de-interlaced imagecomprises: splicing two down-sampled images with an N^(th) resolution toobtain a spliced image with the N^(th) resolution; inputting the N^(th)resolution spliced image into an N^(th) serial de-interleavedsub-network to obtain an N^(th) resolution de-interleaved image;up-sampling an N^(th) resolution de-interlaced image to obtain anN-1^(th) resolution up-sampled image; splicing an up-sampled image withan i^(th) resolution and a down-sampled image with the i^(th) resolutionto obtain a spliced image with the i^(th) resolution; inputting thespliced image with the i^(th) resolution into an i^(th) deinterleavingsub-network to obtain a de-interleaved image with the i^(th) resolution;performing up-sampling processing on the de-interlaced image with thei^(th) resolution to obtain an up-sampled image with an i-1^(th)resolution; wherein i is an integer greater than or equal to 2 and lessthan N; and splicing an up-sampled image with a first resolution and anoriginal video image with the first resolution to obtain a spliced imagewith the first resolution; inputting the first resolution spliced imageinto a first de-interlacing network to obtain a first resolutionde-interlaced image as an output image of the de-interlacing network.21. The electronic device according to claim 13, wherein combining theodd field data and the even field data with the same resolution amongthe odd field data with N-1 different resolutions and the even fielddata with N-1 different resolutions comprises: arranging the odd fielddata and the even field data having the same resolution in lineintervals.